Video analytics system

ABSTRACT

A video analytics system includes a first database for storing searchable time-stamped transactional data indicative of activity within a monitored system, a second database for storing time-stamped video metadata, wherein the time-stamped video metadata comprises searchable attributes associated with a raw video data stream; and a rule-based correlation server for comparing the time-stamped transactional data with the time-stamped video metadata to identify correlation events indicating potential activity of interest. An output subsystem reports the correlation events from the correlation engine. The analytics system is useful for detecting fraud in ATM transactions by comparing the transactional data, for example, the presence of a transaction, with video metadata, for example, indicating whether a transaction occurs when a person is present, for how long the person is there.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims the benefit as a continuation of U.S.patent application 14/117,108 filed Nov. 12, 2013 entitled “VideoAnalytics System” which itself claims the benefit of priority from WorldPatent Application PCT/CA11/00553 filed on May 12, 2011 entitled “VideoAnalytics System” wherein the entire contents of each are hereinincorporated by reference.

FIELD OF THE INVENTION

This invention relates to the field of analytics, and more particularlyto a video analytics system, useful, for example, fraud detection, andmore particularly ATM fraud protection.

BACKGROUND OF THE INVENTION

An automated teller machine (ATM), also known as an automated bankingmachine (ABM) or Cash Machine is a computerized telecommunicationsdevice that provides the clients of a financial institution with accessto financial transactions in a public space without the need for acashier, human clerk or bank teller.

On most modern ATMs, the customer identifies himself by inserting aplastic ATM card with a magnetic stripe or a plastic smart card with achip, which contains a unique card number and some security information.Authentication is achieved by the customer entering a personalidentification number (PIN).

ATMs are placed not only near or inside the premises of banks, but alsoin locations such as shopping centers/malls, airports, grocery stores,petrol/gas stations, restaurants, or any place large numbers of peoplemay gather. These represent two types of ATM installations: on and offpremise. On-premise ATMs are typically more advanced, multi-functionmachines that complement an actual bank branch's capabilities and thusmore expensive. Off-premise machines are deployed by financialinstitutions and also ISOs (or Independent Sales Organizations) wherethere is usually only a need for cash, so they typically are the cheapermono-function devices. In North America, banks often have drive-thrulanes providing access to ATMs.

An ATM typically includes a CPU (to control the user interface andtransaction devices), a magnetic and/or Chip card reader (to identifythe customer), a PIN Pad often manufactured as part of a secureenclosure, a Secure crypto-processor, generally within a secureenclosure, a Display (used by the customer for performing thetransaction), Function key buttons (usually close to the display) or aTouchscreen (used to select the various aspects of the transaction), aRecord Printer (to provide the customer with a record of theirtransaction), a Vault (to store the parts of the machinery requiringrestricted access), and a Housing (for aesthetics and to attach signageto).

Encryption of personal information, required by law in manyjurisdictions, is used to prevent fraud. Sensitive data in ATMtransactions are usually encrypted with DES, but transaction processorsnow usually require the use of Triple DES. Remote Key Loading techniquesmay be used to ensure the secrecy of the initialization of theencryption keys in the ATM. Message Authentication Code (MAC) or PartialMAC may also be used to ensure messages have not been tampered withwhile in transit between the ATM and the financial network.

There are various methods by which criminals attempt to defraud thesystem. Card skimming or card cloning involves the installation of amagnetic card reader over the real ATM's card slot, which is not easilydetectable. The devices used are smaller than a deck of cards, and areused in association with a wireless surveillance camera or a digitalcamera that is hidden to observe the user's PIN. Card data is thencloned onto a second card and the criminal attempts a standard cashwithdrawal. The availability of low-cost commodity wireless cameras andcard readers has made it a relatively simple form of fraud, withcomparatively low risk to the fraudsters. Criminals tend to attachskimming devices either late at night or early in the morning, andduring periods of low traffic. Skimming devices are usually attached fora few hours only because of battery life in the camera. It is estimatedthat globally, financial institutions are losing over a billion dollarsannually to card skimming.

Customers count on ATM security, but with ATM skimming on the rise,customer confidence is threatened. 67% of U.S. adults who use bankingATMs would be likely to switch institutions after an instance of ATMfraud or data breach. It is essential that financial institutions takecorrective measures to ensure banking security.

Rules are usually set by the government or ATM operating body thatdictate what happens when integrity systems fail. Depending on thejurisdiction, a bank may or may not be liable when an attempt is made todispense a customer's money from an ATM and the money either getsoutside of the ATM's vault, or was exposed in a non-secure fashion, orthey are unable to determine the state of the money after a failedtransaction.

In an attempt to stop these practices, countermeasures against cardcloning have been developed by the banking industry, in particular bythe use of smart cards which cannot easily be copied or spoofed byunauthenticated devices, and by attempting to make the outside of theirATMs tamper evident. Older chip-card security systems include the FrenchCarte Bleue, Visa Cash, Mondex, Blue from American Express¹ and EMV ‘96or EMV 3.11. The most actively developed form of smart card security inthe industry today is known as EMV 2000 or EMV 4.x.

EMV is widely used in the UK (Chip and PIN) and other parts of Europe,but when it is not available in a specific area, ATMs must fallback tousing the easy-to-copy magnetic stripe to perform transactions. Thisfallback behavior can be exploited. Card cloning and skimming can bedetected by the implementation of magnetic card reader heads andfirmware that can read a signature embedded in all magnetic stripesduring the card production process. This signature known as a“MagnePrint” or “BluPrint” can be used in conjunction with common twofactor authentication schemes utilized in ATM, debit/retailpoint-of-sale and prepaid card applications.

Another ATM fraud issue is ATM card theft which includes credit cardtrapping and debit card trapping at ATMs. Originating in South Americathis type of ATM fraud has spread globally. Although somewhat replacedin terms of volume by ATM skimming incidents, a re-emergence of cardtrapping has been noticed in regions such as Europe where EMV Chip andPIN cards have increased in circulation.

A Lebanese loop is a device used to commit fraud and identity theft byexploiting automated teller machines (ATMs). Its name comes from itsregular use amongst Lebanese financial crime perpetrators, although ithas now spread to various other international criminal groups. TheLebanese loop is becoming one of the simplest and most widespread formsused to perpetrate ATM fraud by retaining the user's card. In theirsimplest form, Lebanese loops consist of a strip or sleeve of metal orplastic (even something as simple as a strip of video cassette tape)that is inserted into the ATM's card slot. When the victim inserts theirATM card, the loop is sufficiently long enough for the card to be fullydrawn into the machine and read. The victim then enters their PIN asnormal, and requests the funds. The ATM then tries to eject the card,but a “lip” folded at the end of the loop prevents the card from beingejected. The machine senses that the card has not been ejected, anddraws the card back into the machine. The cash drawer does not open, andthe money that has been counted is retained by the machine. In mostcases, the victim's account is not debited. The victim believes themachine has malfunctioned or genuinely retained their card. In somecases, the fraudsters attach a small camera to the ATM to record thevictim entering their PIN. The video from this camera is thentransmitted to the fraudsters, who may be waiting near the machine andviewing the video on a laptop computer meaning they need not approachthe victim directly. There have been cases where a fake keypad is fittedto the machine over the top of the real one, and this records the PINsentered. Once the victim has left the ATM, the perpetrator retrieves theloop and the trapped card, and uses it, along with their PIN, towithdraw cash from the victim's account.

There are different types of cameras used at locations for securitypurposes. One type is expensive, and does video analytics itself, or iscombined with an expensive encoder attached to the camera (the embeddedvideo analytics automatically monitor the video by watching for motiondetection, object recognition and many other security threats). Theother is much less expensive and just takes video, from which images canbe extracted from every set time period. In both cases, the cameras runcontinuously.

Various approaches are currently used to address the problem of ATMfraud.

Diebold sell ATM machines. Their card-skimming technology includes ATMcard-reader security designed to deter skimmer attachment, an alertsystem that warns bank personnel thieves have attached a skimming deviceto an ATM and an electromagnetic field that interferes with a skimmer'sability to capture a card's magnetic-stripe data.

Diebold's monitoring center also issues real-time e-mail alerts and textmessages warning bank employees of skimming attacks.

Customers have to buy their equipment; therefore, it is not a solutionfor installed base.

ADT has CPK+ (Card Protection Kit) technology, an advanced anti-skimmingprotective device installed inside the ATM near the ATM's card reader.CPK+helps prevent the skimming of card data by emitting anelectromagnetic field to interrupt the operation of an illegalcard-reader head, without interrupting the customer transaction or theoperation of most ATMs. They also have Surface Detection Kit (SDK); theSDK sensor helps detect foreign devices placed near or over the ATMcard-entry slot, whether made of plastic, paper, iron or wood. Upondetection, it relays output signals, triggering silent alarms formonitoring center response, or to coordinate DVR surveillance sequencingof skimming activities.

ADT Anti-Skim sensing devices can be integrated with ATM or vestibulesurveillance DVRs for video documentation and sequencing of skimmingactivities and corresponding ATM customer transactions.

ADT also has Monitoring Centers to monitor the ATM security program, andthe customer can receive real-time notification of ATM skimmingoccurrences and law enforcement or security can be dispatched to reviewATMs in alarm or remove detected skimming devices.

ATM Secure has a product Shadow Shield-ECS which offers the ability toprovide an electronic shield in the vicinity of the card reader, thusproviding a jamming protection shield. This prevents any cardreading-skimming device from collecting data, when placed within a 100mmradius of the ATMs card reader. In addition to this, Shadow Shield-ECStransmits a signal that is designed to confuse, and corrupt datacollected by an attached skimmer.

They also have SED-E-field which provides an electronic sensing areaaround the ATM card slot, and can detect the presence of foreign objectslike card skimming devices. Once detected, the SED-E-field can send analarm signal to the security system, alerting of the detection.

Wincor Nixdorf sells ATM machines. They have increased even further thesecurity in and around ATMs with the software solution ProView for theremote monitoring of self-service banking systems. Monitoring ofanti-skimming modules has now been integrated into the bank machine.Anti-skimming modules are equipped with special sensors that check thearea around the card insertion point for illicitly installedattachments. If such a module detects anything suspicious, it sends an“event” to ProView, which immediately initiates a variety of protectivemeasures: for instance, it can activate a camera, photograph theperpetrator, take the ATM offline and generate a report to the serviceprovider. If the camera monitoring an ATM fails, ProView can also takethe machine offline.

They also have an anti-skimming mechanism, which is a plastic insertthat can be mounted in the card reader slot. The shape of the specialinsert is designed to prevent tampering with skimming mechanisms but, atthe same time, does not restrict ATM usage. The anti-skimming mechanismis equipped with security technology that puts the machine out ofservice as soon as the insert is destroyed, or the machine removed byforce.

Customers have to buy their equipment; therefore, it is not a solutionfor installed base.

Jitter technology works via a stop start or jitter motion inside thecard drive specifically designed to distort the magnetic stripe detailsshould they be copied onto a foreign card reader inserted into the ATM.

Video Analytics, also known as IVS (Intelligent Video Surveillance) is anew emerging market for security allowing its users to easily monitorand secure areas with security cameras. With this new state of the arttechnology, businesses can easily monitor places of interest withsophisticated software that makes detecting threats or unwanted visitorssimple and effective.

Intelligent Video Surveillance consists of algorithms that detectmovement or changes in live and recorded video to see whether themovement or changes mean a possible threat is about to occur oroccurring. These algorithms work by examining each pixel of the videoand putting together all the pixel changes. If many pixels are changingin one area and that area is moving in a direction, the softwareconsiders this to be motion. Depending on the policies and alerts thathave been setup, the bank will be notified of this motion. Other actionscan be automatically taken by the as motion tracking which follows themotion until it is no longer detected. It can include LoiteringDetection, Queue Length Monitoring and Facial Detection, among otherthings.

There are various problems with current solutions. Jitter is a securityfeature, but it helps only for simple skimmers. With motorized skimmersor extended skimmers, only a sensory solution will offer protectionbecause magnetic stripe data still can be read.

Sensor detection does not work well, because it can be set off by acustomer's electronic device like a cell phone or iPod.

New video cameras are very expensive, and can be prohibitive from a costpoint of view given the number of ATMs that would have to be fitted withthe cameras.

No current solution targets cash harvesting, which occurs when thethieves take the money out using the fake cards. No current solution canwarn of skimmer installation.

US patent application no. 2008/0303902 describes a system for collectingvideo data and transactional data and correlating the two. However, thissystem is bandwidth intensive because the video data is processed alongwith the transactional data, and is thus not suitable for large-scalesystems.

SUMMARY OF THE INVENTION

According to the present invention there is provided a video analyticssystem comprising

a first database for storing searchable time-stamped transactional dataindicative of activity within a monitored system; a second database forstoring time-stamped video metadata; a correlation server for comparingthe time-stamped transactional data with the time-stamped video metadatato identify correlation events indicating potential activity ofinterest; and an output subsystem for reporting the correlation eventsfrom the correlation server.

In one embodiment the time-stamped video metadata comprises searchableattributes associated with a raw video data stream, but it could also bedata indicative of a customer session, where the customer is detected byanalyzing the frames of the video to detect the presence of a person orjust mere motion in the video image.

It will be appreciated that the first and second databases could bephysically separate entities, or could in the alternative equally wellbe part of the same physical storage medium.

A feature of invention is the conversion of the raw video data intosearchable, time-stamped metadata, which can then be stored andcorrelated without consuming large amounts of bandwidth. A large numberof facilities can be monitored at a central location without consumingunmanageable amounts of bandwidth. For example, in the case of an ATMmachine, the metadata may indicate the presence of a person in front ofthe machine. This can then be correlated against the transaction data toconfirm that a transaction did take place. If there was no transaction,the raw video data, or frames thereof, can be view to identify theperson or see whether there were performing an illegal activity, such asinstalling a skimming device.

The output subsystem may be a correlation database for storing thecorrelated data for subsequent review. It could also be part of amonitoring station for attracting the attention of a supervisor.

The invention is particularly applicable to ATM fraud detection, butthis is just one example of a more general embodiment of the inventionthat takes two or more independent data streams, time stamped data andtime stamped video, and correlates them using time and location and runsan analysis to detect when relevant information is present. Thiscorrelation and analysis can be applied to other situations as well. Itcould be done at a point of sale terminal, when a badge is swiped, whena door is opened or closed, when an area (like a store) is open orclosed, when a traffic light is red, green or yellow, when a fire alarmis pulled, etc., or any time there is video data and other timesensitive data that is being stored at the same time. In accordance withembodiments of the invention, by correlating the video metadataextracted from the raw video with other transactional data avoids theneed to correlate the transactional data directly with the raw video.

Embodiments of the invention thus provide an advanced data analyticsystem that uses video/picture metadata and time sensitive data (whichcould be ATM transactions, badge swiping, retail transactions, trafficlight schedules, operating hours, crime patterns, emergency situations,etc.) and uses customizable rules to detect a predefined activity. Ituses an algorithm or correlation engine to do a proactive analysis, andwhen a possible problem is found, it can alert the appropriate person(s)to the activity discovered.

The video can be in the form of metadata obtained from embeddedanalytics created by a camera and stored on site in a DVR. Suchintelligent cameras are however expensive and if not already in placewould need to be specially installed. Embodiments of the invention alsoprovide a much more economical solution, which is to take existing videoshot from normal cameras, and create the metadata at a centralizedlocation. No software has to reside locally where the cameras are, andall processing and storage is done centrally. This then creates asearchable record of the meta-data taken from the images (license plate,height, colors, facial features etc.). If a company is already streamingvideo to a location like a security room (and the video is not storedanywhere), then a ‘filter’ could be put on this stream which can extractanalytics or images on the fly and send them to the centralizedlocation. Another way of conserving bandwidth is to have a trigger, suchas motion detection or a transaction which tells the system to captureimages for a particular time frame to determine the start and stop timefor the session.

Embodiments of the invention can, for example, be used to determiningthat there is someone at an ATM with no corresponding transaction,suggesting a possible skimmer installation, or determine that the sameperson did many transactions in a row at the same ATM using differentcards, suggesting possible cash harvesting. Another application is todetermine that there is an unauthorized person present in a buildingduring off hours by using the guard schedule (either pre-programmed, ordone in real time by badge swiping in different areas), suggesting apossible break-in. Further applications include determining that thesame license plate is seen at many different ATM branches, suggestingpossible cash harvesting, determining that after a badge swipe, thewrong person or the wrong number of people are in an area, suggestingpossible theft or fraud, determining that the same person used a stolencard at one or more locations, suggesting possible theft.

Also, if it is known that a crime took place at a certaindate/time/place, video analytics taken from different cameras in thearea can be used to track the person, or for example a hit and run in aparking lot, to determine the license plate. Using video analytics froma subway station combined with the subway schedule to determine if thereis enough capacity on the trains at various times of the day.

The solution lets the user configure at least one variable thatinfluences whether an activity is to be flagged or not. This can be donethrough a user interface, or an Application Programming Interface (API).

It also can display the results in a unique way making it very easy fora user to find significant instances.

Embodiments of the invention correlate video and data evidence thatmight otherwise take days to collect, and present it to a customizableset of users in an organized dashboard. This enables users to, forexample, detect fraud crimes quickly, investigate them easily, andultimately reduce the losses they cause. Once they are alerted, userscan drill down into the data and retrieve all the correlated andrelevant information. In the case of fraud detection, this dramaticallyimproves investigation capabilities, reduces the time and cost perinvestigation, and produces superior evidence for prosecution purposes.

There is no need for local hardware at each site; therefore, there areno protocol or encryption issues.

Embodiments of the invention permit the user to search network-wide on abroad range of customized transaction data fields, use simple refinementtools to drill down into relevant video and data, combine desired dataand synchronized video into compelling files with interactive casenotes, and export video clips, images, case notes and/or receipt datafor easy use by enforcement personnel.

Another aspect of the invention provides a method of monitoring a sitefor activity of interest, comprising storing searchable time-stampedtransactional data indicative of activity at the site; storingtime-stamped video metadata; and performing a rule-based correlationserver of the time-stamped transactional data with the time-stampedvideo metadata to identify correlation events indicating potentialactivity of interest, and reporting the correlation events.

In yet another aspect the invention provides a video analytics systemcomprising a video analytics system comprising a first database forstoring searchable time-stamped transactional data indicative ofactivity within a monitored system; a second database for storingtime-stamped metadata indicative of the presence of a person at alocation monitored by a video camera producing a raw video stream; acorrelation server for comparing the time-stamped transactional datawith the time-stamped metadata to identify correlation events indicatingpotential activity of interest; and a module for analyzing video datacorresponding to a correlation event.

In this embodiment, the analytics are triggered by, for example, thedetection of a person in the view of the camera. In this embodiment thesystem detects a person and then determines whether there is a matchingtransaction.

In one embodiment the video analytics module is configured to identifythe presence of a person by carrying out the following steps:

-   -   retrieve any available data with a time-stamp;    -   retrieve one or more video frames corresponding to the        time-stamp;    -   analyze the retrieved frames to identify correlation events; and    -   store the retrieved video frames or raw video stream associated        with correlation events.

This embodiment of the invention also conserves bandwidth because theperson detection is performed on individual frames rather than the videostream.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described in more detail, by way of exampleonly, with reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of a video analytics system in accordance withone embodiment of the invention;

FIG. 2 is a more detailed block diagram of the correlation server;

FIG. 3 is a flow diagram illustrating operation of the system;

FIG. 4 is a block diagram of the correlation server;

FIG. 5 is the pseudo code for a skimming installation module;

FIG. 6 is the pseudo code for a person-based harvesting detectionmodule;

FIG. 7 is the pseudo code for a license-based harvesting detectionmodule;

FIG. 8 is the pseudo code for an unauthorized access module;

FIG. 9 is the pseudo code for a capacity detection pseudo code module;

FIG. 10 is the pseudo code for a pinpoint detection module;

FIG. 11 is the pseudo code for a perpetrator detection module;

FIG. 12 is the pseudo code for a vagrant on premises detection module;and

FIG. 13 is a flow chart of an exemplary bandwidth-efficient embodimentfor creating a session to perform video analytics.

DETAILED DESCRIPTION OF THE INVENTION:

The invention will be described by reference to one exemplary use of thevideo analytics system, which is for ATM fraud detection.

In this embodiment, the ATM video and transaction records are convertedinto searchable data. This can be used to detect people approaching andloitering at an ATM, capture total time at ATM facility, target skimmingdevice installation or removal, detect multiple card use per session asindicator of cash harvesting, and detect cars being used at more thanone ATM.

Investigator performance can be improved by filtering data daily to flagall new threats automatically, consolidating all data and video eventsin a single, organized dashboard, creating notification of alarms.

Any bank or credit union is able to providing customizable alarms thatcan be tailored to detect the specific behaviors each institutionrequires, allowing new rules to be tested against historical data tomore quickly uncover new fraud patterns, and solve a cost problem ofexpensive cameras that have video analytics by providing a lower costsoftware solution that does the same thing.

Referring now to FIG. 1, the system contains multiple time sensitiveservers 107 connected to the LAN/WAN 105 that contain time sensitivedata, such as ATM transactions, or guard schedules, or train/subwayschedules. These store the information in Databases 108, and provide anAPI 110 to access this data. They can reside in a Branch Office 117, orin a Cloud Service or Head Office 118.

Another source of data comes from Video Data Servers 103 connected tothe LAN/WAN 105. These servers provide access to raw video data or videoanalytics data, derived from the raw video data as searchable metadata,stored in Raw Video/Analytics Databases 102. They can reside in a BranchOffice 117, or in a Cloud Service or Head Office 118.

The Correlation Server 100 contains Crime Detection Software 101,responsible for using the time sensitive data and the video metadata todetermine if there is fraudulent activity in accordance with apredetermined, but configurable, set of rules. It can also have a remotefunction running on a Remote Correlation Server 105, which can run theVideo Analytics Creation Module 211 to create video analytics from rawvideo. If fraudulent activity is found, it stores the information incorrelation Database 104. Database 104 also contains any imagereferences, a copy of the time sensitive data, and the video and videoanalytics (which can come from the Video Data Server 103, or could havebeen created by the Remote Correlation Server 105 from raw video).

The video metadata or video analytics is searchable time-stamped dataassociated with the video as well as other information about the video.For example, the video analytics may detect a person present in front ofan ATM machine. The video metadata includes this searchable tag, whichmay be stored in association with an image of the person. Also, facerecognition software might be used to tag the person with a particularidentity, either known or unknown. The latter might be useful, forexample, to determine that the same person withdrew cash from differentmachines within a specified time frame, as this could indicate cashharvesting.

The metadata can be derived from the video either at the source, using amore expensive camera, or by the system, for example, using thecorrelation server. To avoid using up unnecessary bandwidth, in ascenario where a bank branch has an installed base of dumb cameras, theanalytics could be done on the local network at the branch so that onlythe metadata need to be sent to a central monitoring or analysisstation. Alternatively, the raw video could be sent to the centralmonitoring station for analysis, but this would require more bandwidth.

Communications Servers 106 are also connected to the LAN/WAN 105, whichprovide a standard API 113 that lets the Correlation Server 100 handleany communication that is sent to a programmable set of interestedparties when an incident occurs. Examples of communications sent areemail, SMS, voice calls, tweets, chats, pages, etc.

Users 112 can access the data in Database 104, set up parameters for thesystem, configure the rules for determining the fraudulent activity andcause analysis to be done by using web pages 111 that are connectedthrough the LAN/Wan 105 to the Correlation Server 100. The User 112 canalso view incidents, search incidents, view the video associated withthe incidents, view the data associated with the incidents, and addnotes to the incidents. These APIs are implemented using knowntechniques.

FIG. 2 is a more detailed view of the Correlation Server 100. Thisincludes a Time Sensitive Data Interface module 200, which handles thedifferent APIs 110 that are used by the Time Sensitive Data Servers 107.The correlation server 100 also includes a Video Data Interface module201, which handles the different APIs 109 that are used by the VideoData Servers 103. There is a Communications Server Interface module 202,which handles the different APIs 113 that are used by the CommunicationsServers 107. The correlation server also includes video data interface203, which interfaces with video data servers 103. These APIs alsoprovide access to the data in Raw Video/Analytics Database 102. Thesecomponents all managed by the Control Layer 210.

This control layer is responsible for coordinating all the activity thatgoes on within the correlation server. The Video Analytics CreationModule 211 uses video analytics algorithms to extract data from a givenstandard video or set of images. It can also be asked by the Get,Decompose and Store Video Data module 301 (see FIG. 3) to analyze thevideo frame by frame as shown in FIG. 13. This reduces bandwidth sincewhen it is done, there is no need to download the video, just a coupleof frames and the analytic information on what happened during thedetermined time period. This can be done locally, or it can reside on aremote server and is accessed through a Video Analytics Creation ModuleInterface 207. There is a Web Interface 203 which talks to Web Pages 111in a standard way, for example using SOAP 205.

There is a Correlation Database Interface 204, which uses a standarddatabase API 206 to get and store data in the Correlation Database 104.There are a set of pluggable Correlation Modules 213, which does thework of finding incidents and reporting on them.

FIG. 3 shows a Get, Decompose and Store Time Sensitive Data module 300,which when it is told to do so by the Timer/Capacity Queue 302, it isresponsible for interfacing to the Time Sensitive Data Interface 208 toget time sensitive data for a particular time frame, break it up intodifferent records, and store these records in the Correlation Database104, by using the Write/Get Data and Records module 305.

The Get, Decompose and Store Video Data module 301, when it is told todo so by the Timer/Capacity Queue 302, is responsible for interfacing tothe Video Data Interface 208 to get video data for a particular timeframe, (and at a specific frame/sec rate in the case of video—toconserve bandwidth), break it up into different records, and store theserecords in the Correlation Database 104, by using the Write/Get Data andRecords module 305. If it is retrieving straight video, it takes thevideo and passes it to the Video Creation Module 211, which usesstandard techniques to extract video analytics from the passed in data,which are given back to be stored in the database.

The Timer/Capacity Queue 302 is also responsible for deciding when tokick the Correlation Module Kicker 303, based on either time, or acertain capacity of data being reached. It keeps track of whichcorrelation modules need to be kicked, and what the trigger is.

The Correlation Module Kicker 303 is used to kick or activate thevarious Correlation Modules 213. It is told to do this either by theTimer/Capacity Queue 302, by the Web Interface 203 (when a User 112decides they want to), or the Video Data Interface 201, which can beprogrammed to receive a motion trigger, which in turn can kick theCorrelation Module Kicker 303.

When a User 112 wants to access the data in the Correlation Database104, the Web Interface 203 uses the Write/Get Data and Records module305 which in turn talks to the Correlation Database Interface 204 toretrieve the data. The Web Interface 203 is responsible for formattingthe data and sending it to the Web Pages 111. The user can also requestto have a particular video analyzed, and certain characteristicssearched for. The Web Interface 203 kicks the Get, Decompose and StoreVideo Data module 301 to do this.

The Write/Get Data and Records module 305 gets requests to store dataand records from the Get, Decompose and Store Time Sensitive Data module300, and the Get, Decompose and Store Video Data module 301. Itretrieves data for the Web Interface module 203 and the CorrelationModules 213. The Correlation Modules 213 also use it to store recordsthat they create.

The Notify module 304 is used by the Correlation Modules 213 to talk tothe Communications Server Interface to send out various types ofcommunications to a programmable group of people.

FIG. 4 shows the pluggable Correlation Modules 213 broken down into someexample modules. The Control Layer Interface 400 passes the variousrequests to and from the different plug-in modules. These correlationmodules use helper modules that perform standard functions. The helpermodules are the Notification Module 408, which has stored the group ofpeople to be notified, and how they should be notified, the RequestFurther Analytics Module 409, which can request that a video for aparticular time period be retrieved, and further analytics be done on itby the Video Creation Module 211, and the Create and Store IncidentReport Module 410, which knows how to format the information given to itinto an incident report and have it stored.

The Skimming Installation Module 401 uses the data it retrieves todetermine if a skimmer has been installed at a card reader site.

The Unauthorized Access Module 402 uses the data to determine if thereis an unauthorized person or persons present.

The Capacity Detection Module 403 uses the data to determine if enoughpeople are being serviced in a particular time period.

The Vagrant on Premises Module 404 determines if there is a personstaying in one spot for a period of time (i.e. someone sleeping).

The Harvesting Detection Module 405 uses the data to see if the sameperson, or the same license plate does more than one transaction in arow using the same card reader, but different cards.

The Perpetrator Detection Module 406 correlates the use of a stolen cardwith videos obtained from different sites.

The Pinpoint Detection Module 407 uses the area where a crime occurred,the time that it occurred, and the video from different cameras to do ananalysis to try and pinpoint the perpetrator.

Other correlation modules can be created and plugged in that use thedata available to come to a conclusion and produce a report.

FIGS. 5 to 12 give examples of pseudo code executed by the modules 401to 406 in FIG. 4.

FIG. 13 illustrates one example of video analytics that might beperformed on a video stream by creation module 211 in FIG. 2.

At step 1300, a trigger occurs at time T to get an image from the videoat step 1301. At step 1302 the image is analyzed, and a determinationmade (step 1303) whether a person is in the image at time T-x, where xis a predefined time period. If not, the process terminates 1304. Ifyes, at step 1305, the creation module retrieves an image from the videoand at time T+x and analyzes the video at step 1306 to determine ifthere is a different person in the image at step 1307. If the person isthe same, the loop repeats from step 1305 at a later time T+2x relativeto the original time T and so on. If there is a different person in theimage, the loop passes to step 1309, which sets the analysis time at anintermediate time between the first and second steps. The loop repeatsuntil no person is found in the image at step 1310.

This process thus identifies the fact that a person was present at theATM and for how long. This metadata can be sent to the video databaseand stored along with the relevant images and a link to the associatedvideo. A monitoring station can be notified by the communications server106 in the event that a person lingers for an unusual amount of time atan ATM machine or is found at several different ATMs within a shorttime-frame.

ATM fraud detection is just an example of taking two or more independentdata streams, time stamped data and time stamped video, and correlatingthem using time and location and running an analysis to detect whenrelevant information is present. This correlation and analysis can beapplied to other situations as well. It could be done at a point of saleterminal, when a badge is swiped, when a door is opened or closed, whenan area (like a store) is open or closed, when a traffic light is red,green or yellow, when a fire alarm is pulled, etc. Any time there isvideo data and other time sensitive data that is being stored at thesame time.

Advantages of described system include the fact that it can be used onany installed system as there is no need to change the data collectorand cameras, it combines and correlates and analyzes video analyticswith transaction software that is better than just video or data alone.,it can be integrated with existing CCTV/Analog video systems orimplemented with new state of the art IP network camera, fewer personnelare needed to view video. With standard video systems, someone mustalways be watching. This decreases labor costs and increasesproductivity.

The system also capitalizes on existing video analytics data andcombines it with transaction data to expose possible fraud, notdetectable by the two individually. It can be used to detect unwantedpeople in an area, not needing special equipment, just a video camera.It can also be offered as a central solution inside a company that hasmany video cameras, or as a cloud solution.

It should be appreciated by those skilled in the art that any blockdiagrams herein represent conceptual views of illustrative circuitryembodying the principles of the invention. For example, a processor maybe provided through the use of dedicated hardware as well as hardwarecapable of executing software in association with appropriate software.When provided by a processor, the functions may be provided by a singlededicated processor, by a single shared processor, or by a plurality ofindividual processors, some of which may be shared. Moreover, explicituse of the term “processor” should not be construed to refer exclusivelyto hardware capable of executing software, and may implicitly include,without limitation, digital signal processor (DSP) hardware, networkprocessor, application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA), read only memory (ROM) for storingsoftware, random access memory (RAM), and non-volatile storage. Otherhardware, conventional and/or custom, may also be included. The termcircuit is used herein to encompass functional blocks that may inpractice be implemented in software.

What is claimed is:
 1. An analytical system comprising: a local systemassociated with a physical location comprising a first microprocessor, afirst network interface connecting the local system to a communicationsnetwork, a first interface for acquiring visual data relating to apredetermined region associated with a system to be monitored (monitoredsystem), and a second interface for acquiring system data from themonitored system; wherein the local system acquires visual data relatingto the monitored system and processes it to generate searchabletime-stamped video metadata relating to the predetermined regionassociated with the monitored system; the local system acquires systemdata relating to the monitored system and processes it to generatesearchable time-stamped system metadata relating to the monitoredsystem; and the local system transmits the searchable time-stamped videometadata and searchable time-stamped system metadata to either a remoteserver or a plurality of data servers via the first network interface;the remote server comprising a second microprocessor, a second networkinterface connecting the remote server to the communications network, athird interface connecting the remote server to an output system, and afirst memory, wherein the remote server stores the searchabletime-stamped system metadata relating a monitored system within a firstdatabase within the memory; the remote server stores the time-stampedvideo metadata relating to a predetermined region associated with themonitored system within a second database within the first memory; theremote server processes the time-stamped system metadata and thetime-stamped video metadata to identify correlation events indicatingpotential activity of interest in dependence upon a predetermined set ofconfigurable rules, each configurable rule relating to detection of apredefined activity within the first memory; and the remote serverprovides data relating to identified correlation events to the outputsystem; and the output subsystem comprising a third microprocessor, afourth interface connecting the output system to the remote server, anda second memory, wherein the output system stores within a thirddatabase within the second memory the data relating to identifiedcorrelation events; the output system generates for each identifiedcorrelation event a web page comprising the data relating to theidentified correlation event; the output system generates a message fortransmittal to a predetermined user comprising at least a link to theweb page; wherein the predetermined user is established in dependenceupon the identified correlation event and an aspect of the identifiedcorrelation event the correlation server processes the time-stampedvideo metadata as searchable attributes associated with a raw video datastream acquired by the local system which have been generated by thelocal system by analyzing the raw video stream in accordance with astored set of rules.
 2. The analytical system according to claim 1,wherein the local system transmits the visual data relating to themonitored system to the remote server via the first communicationsinterface; the remote server stores the received visual data from thelocal system within the first memory; the remote server transmits one ormore frames of the received visual data or raw received visual data tothe output system, the one or more frames of the received visual data orraw received visual data established by the remote server in dependenceupon the identified correlation event; the output system stores inassociation with the identified correlation event the one or more framesof the received visual data or raw received visual data; and the outputsystem includes the one or more frames of the received visual data orraw received visual data within the generated webpage associated withthe identified correlation event.
 3. The analytical system according toclaim 1, wherein where the local system transmits the searchabletime-stamped video metadata and searchable time-stamped system metadatato a plurality of data servers via the first network interface the localsystem: transmits the time-stamped system metadata relating to themonitored system to a first time-sensitive data server at a first remotelocation via the first network interface; and the time stamped videometadata relating to the monitored system to a second time-sensitivedata server at a second remote location via the first network interface;and the remote server: retrieves from the first time-sensitive dataserver at the first remote location the time-stamped system metadata;and retrieves the time-stamped video metadata from the secondtime-sensitive data server at a second remote location.
 4. Theanalytical system according to claim 1, wherein the remote server atleast one of: extracts from the time-stamped video metadata dataindicative of the presence of a person at a monitored location;processes the time-stamped video metadata data to determine whether aninstance of a person being present at the monitored location hasoccurred; performs a search backwards in time from a time-stampassociated with the presence of a person to establish an approximatearrival time of the person within the region monitored; performs asearch forwards in time from a time-stamp associated with the presenceof a person to establish an approximate departure time of the personwithin the region monitored; and requests additional at least one oftime-stamped system metadata and time-stamped video metadata in responseto an indeterminate correlation event.
 5. The analytical systemaccording to claim 1, wherein the remote server exploits a plurality ofcorrelation modules, each correlation module for establishing acorrelation event selected from the group comprising a financialinstrument skimming installation, a financial instrument harvestingactivity, an unauthorized access, an unauthorized presence, atransportation capacity failure, presence of a crime perpetrator, and aphysical location determination.
 6. A method comprising: providing anoutput system for transmitting to a predetermined user a messagecomprising at least a link to a web page, the output system:establishing the predetermined user is established by the output systemin dependence upon an identified correlation event and an aspect of theidentified correlation event; and generating for each correlation eventa web page comprising information relating to the identified correlationevent; wherein the occurrence of the identified correlation event iscommunicated to the output system by a correlation server incommunication with the output system via a communications network andthe information relating to the identified correlation is stored withina database by the correlation server accessible to the output system;and the correlation server establishes an occurrence of a correlationevent by executing a process comprising: generating a trigger independence upon a physical event relating to a monitored system;accessing first searchable time-sensitive metadata relating to themonitored system established over a first period of time in dependenceupon the trigger from the trigger generator; accessing second searchabletime-sensitive metadata relating to multimedia image content establishedover a second period of time relating to monitoring a predeterminedregion associated with the monitored system in dependence upon thetrigger from the trigger generator; determining in dependence upon apredetermined set of configurable rules whether a correlation eventexists by processing the first searchable time-sensitive metadata andsecond searchable time-sensitive metadata for processing thetime-stamped system metadata to identify correlation events indicatingpotential activity of interest; storing upon determination of acorrelation event within the database correlation data, the correlationdata comprising at least a predetermined portion of the first searchabletime-sensitive metadata and a predetermined portion of the secondsearchable time-sensitive metadata; wherein each configurable rulewithin the predetermined set of configurable rules relates to detectionof a predefined activity; the correlation server processes the secondsearchable time-sensitive metadata as searchable attributes associatedwith a raw multimedia data stream which have been generated by analyzingthe raw multimedia stream in accordance with a stored set of rulesrelating to at least one of the potential activity of interest and thetrigger generator; the output subsystem stores in association with acorrelation event one or more corresponding video frames orcorresponding raw video is stored with a correlation event; informationrelating to the correlation event within the generated webpage includesthe one or more corresponding video frames or corresponding raw videostored with the correlation event; and the first searchabletime-sensitive metadata relates to a system within a predeterminedlocation accessible by a user and the second searchable time-sensitivemetadata relates to multimedia content acquired within a predeterminedregion associated with the predetermined location.
 7. The methodaccording to claim 6, wherein the correlation server at least one of:extracts data indicative of the presence of a person at a monitoredlocation from at least a predetermined portion of the first searchabletime-sensitive metadata and a predetermined portion of the secondsearchable time-sensitive metadata; processes the at least apredetermined portion of the first searchable time-sensitive metadataand a predetermined portion of the second searchable time-sensitivemetadata to determine whether an instance of a person being present atthe monitored location has occurred; performs a search backwards in timefrom a time-stamp associated with the presence of a person to establishan approximate arrival time of the person within the region monitored;performs a search forwards in time from a time-stamp associated with thepresence of a person to establish an approximate departure time of theperson within the region monitored; and requests additional at least oneof the at least a predetermined portion of the first searchabletime-sensitive metadata and a predetermined portion of the secondsearchable time-sensitive metadata in response to an indeterminatecorrelation event.
 8. The method according to claim 6, wherein thecorrelation server exploits a plurality of correlation modules, eachcorrelation module for establishing a correlation event selected fromthe group comprising a financial instrument skimming installation, anunauthorized access, an unauthorized presence a financial instrumentharvesting activity, a transportation capacity failure, presence of acrime perpetrator, and a physical location determination.
 9. The methodaccording to claim 6, wherein the correlation event is determined tooccur based upon at least one of: the processing of the secondsearchable time-sensitive metadata absent a trigger from the triggergenerator; and processing of the second searchable time-sensitivemetadata together with other second searchable time-sensitive metadatafrom other monitored systems absent triggers from a predeterminedportion of the trigger generators associated with the other monitoredsystems.
 10. A correlation server comprising: a first interface forretrieving first searchable time-sensitive metadata relating to aplurality of monitored systems; a second interface for retrieving secondsearchable time-sensitive metadata relating to multimedia image contentgenerated by monitoring a plurality of predetermined regions, eachpredetermined region of the plurality of regions associated with amonitored system of the plurality of monitored systems; a rule-basedcorrelation module for processing the first searchable time-sensitivemetadata and second searchable time-sensitive metadata to identifycorrelation events indicating a activity of interest, the rule-basedcorrelation module applying a predetermined set of configurable rules;and an output interface for generating for each identified correlationevent a web page comprising information relating to the correlationevent and a message for transmittal to a predetermined user comprisingat least a link to the web page; wherein the predetermined user isestablished in dependence upon the identified correlation event byrule-based correlation module and an aspect of the identifiedcorrelation; each configurable rule within the predetermined set ofconfigurable rules relates to detection of a predefined activity; thefirst searchable time-sensitive metadata and the second searchabletime-sensitive metadata relate to a period of time within which theactivity of interest occurred; and the predetermined set of configurablerules relate to establishing the rule-base correlation module fordetecting the activity of interest.
 11. The correlation server accordingto claim 10, wherein the correlation module relates to establishing acorrelation event selected from the group comprising a financialinstrument skimming installation, a financial instrument harvestingactivity, a transportation capacity failure, a presence of a crimeperpetrator, and a physical location determination.
 12. The correlationserver according to claim 10, wherein the correlation event isdetermined to occur based upon at least one of: the processing of thesecond searchable time-sensitive metadata absent a trigger from thetrigger generator; and processing of the second searchabletime-sensitive metadata together with other second searchabletime-sensitive metadata from other monitored systems absent triggersfrom a predetermined portion of the trigger generators associated withthe other monitored systems.
 13. The method according to claim 10,wherein the first searchable time-sensitive metadata and secondsearchable time-sensitive metadata were previously generated andarchived based upon absence of the correlation event being identified bythe previous application of one or more previously establishedrule-based correlation modules.